feat: add storage tier change events script#118
Merged
Conversation
3 tasks
Collaborator
Author
emmanuelmathot
approved these changes
Apr 2, 2026
The batch storage-tier submitter only windowed on sensing `datetime`, so it
could not target the S2 backlog by *registration* time (`properties.created`).
Bulk-converted items were registered long after they were sensed, so a
sensing-date window misses them.
Add `--date-field {datetime,created,updated}` (default `datetime`, backward
compatible). Non-default fields swap the `datetime=` range for a CQL2 `between`
filter on the chosen property — the same pattern `scripts/query_stac.py` uses
to harvest by `updated`.
Also:
- flip `--storage-class` default `STANDARD_IA` -> `STANDARD` (STANDARD_IA is
unused in the deployment; documented commands stay explicit regardless);
- add `timeout=30` to the webhook POST so a hung endpoint can't stall the
per-window submission loop (ruff S113, on this script's own code);
- document the registered-window backlog example in operator-tools/README.md.
Plan: claude-docs/plans/s2_tier_standard_backlog.md (Task 2)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5997970 to
bc65a8a
Compare
Merged
lhoupert
added a commit
that referenced
this pull request
Jul 13, 2026
Brings in the two commits exclusive to main: #118 (1639baf) storage-tier change-events script #278 (017898d) eodash layerExclusive + style link for S2 L2A Conflicts resolved toward the integration branch (phase5 = source of truth): - operator-tools/README.md tool table: kept phase5's 'uv run python' modernization for the shared rows; added #118's three submit_* rows (normalized to 'uv run python'). - tests/test_stac_collections.py (add/add): unioned — kept phase5's S1 GRD RTC collection asset tests AND main's S2 L2A eodash metadata tests (no function-name overlap). Net change vs phase5 == exactly #118 + #278; no phase5 work reverted. 806 passed, ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
branched from #103 (now rebased on
main; the #103 CLI-command commit is merged, so this PR is just the submitter script + the--date-fieldextension below)submit_storage_tier_workflows.pyoperator tool that queries STAC in 24h windows over a date range and submits onebatch-change-storage-tierwebhook payload per window to trigger theeopf-storage-tier-batch-jobWorkflowTemplate deployed in https://github.com/EOPF-Explorer/platform-deploy/pull/130 via Argo Eventsoperator-tools/README.mdwith full usage docs and a tool-selection reference tableUpdate:
--date-field(window on registration time)The submitter originally windowed only on sensing
datetime. To drain the S2 backlog by registration time (bulk-converted items were registered long after they were sensed), this adds:--date-field {datetime,created,updated}(defaultdatetime, backward compatible). Non-default fields swap the pystac-clientdatetime=range for a CQL2betweenfilter on the chosen property — same patternscripts/query_stac.pyuses to harvest byupdated.--storage-classdefault flippedSTANDARD_IA→STANDARD(STANDARD_IA is unused in the deployment; documented commands stay explicit).timeout=30on the webhook POST (ruff S113, on this script's own code) so a hung endpoint can't stall the per-window loop.Planned use (#182 follow-up): a one-off prod run over
sentinel-2-l2a, windowing on--date-field createdfor items registered2025-11-01 → (today − 3 months), moving their S3 objects to STANDARD. Dry-run first, then live via port-forward. The Sensor + batch template are date-agnostic (they consume resolveditem_ids), so no cluster-side change is needed for this capability.Why
Changing S3 storage classes for thousands of items over a multi-month date range is too slow when done sequentially on a local machine. The script fans work out to Argo: one workflow per 24h window, with intra-window parallelism handled by the
eopf-storage-tier-batch-jobtemplate. Empty windows are skipped automatically. Windowing oncreatedlets us target the registration backlog that a sensing-date window would miss.Test plan
uv run --group test pytest tests/unit/test_submit_storage_tier_workflows.pypasses (incl. new--date-field/ CQL2 / default-storage-class cases)uv run --group test pytest tests/unit/test_change_storage_tier_commands.pypasses--date-field createdbuilds a CQL2betweenoncreated; defaultdatetimepath unchanged (asserted via mockedClient.searchkwargs)action: batch-change-storage-tierwith item IDs per window